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(54) Teaching model generating device 

(57) A workpiece W serving as an object for detec- 
tion is fixed in place. A camera 20 is mounted to the end 
of a robot RB. The camera is turned about an axis, which 
passes through the center position of the workpiece W 
and is perpendicular to the optical axis of the camera, 
to take an image of the workpiece W at a plurality of 



positions in different directions. A teaching model is gen- 
erated on the basis of each produced image data. The 
relative position and posture of the workpiece to the 
camera 20 is also stored in association with the teaching 
model. Thus, it is possible to easily generate the teach- 
ing model of the workpiece regardless of three<)imen- 
sional variations in posture. 



FIG.3 
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Description 

[0001] This invention relates to a device for generat- 
ing a teaching model applied to a method of detecting 
the position and posture of an object through pattern 
matching. More particularly the present invention re- 
lates to a device for generating a teaching model for 
making it possible to detect an object regardless of 
three-dimensional variations in position and posture of 
the object for detection. 

[0002] with a well-known method and device for de- 
tecting the position and posture of an object set on a 
plane by storing a two-dimensional image of the object 
for detection is stored as a teaching model, prior to pat- 
tern mataching of the teaching model with image data 
produced by taking an image of the object with an image 
pickup means such as a camera. The above method 
and device for detecting the object is applied to convey- 
ing or like jobs, of machine parts and articles or like 
workpieces grasped with a robot, to a predetermined po- 
sition in an automatic mode. 

[0003] However, picking-out of an individual work- 
piece from a pile of disordered same-shaped workpiec- 
es or a set of same-shaped workpieces housed within 
a predetermined range in any three-dimensionally dif- 
ferent position and posture is not suited for the robot, 
and therefore, requires manual operation. In the case of 
allowing the robot to pick out such an individual work- 
piece, there is the need for the manual operation of pick- 
ing out the workpieces one by one Irom the pile of dis- 
ordered workpieces for reoriented alignment such that 
the robot may pick out the individual workpiece directly 
[0004] For allowing the robot to grasp each workpiece 
for picking out from a large number of same-shaped 
workpieces piled up disorderly or housed within the pre- 
determined range in any three-dimensionally different 
position and posture, it is necessary for the robot to rec- 
ognize the position and posture (direction) of each of the 
above workpieces. 

[0005] In this connection, a device for detecting the 
three-dimensional position and posture (direction) of the 
individual workpiece among the workpieces has been 
developed. This device performs the following opera- 
tions. 

(1) To produce image data in advance by taking an 
image of one of same-shaped workpieces for de- 
tection with a camera from different directions 

(2) To store a plurality of teaching models based on 
the produced image data, together with the relative 
position and posture of each teaching model to the 
workpiece 

(3) To select a teaching model of a high matching 
value by carrying out pattern matching of each 
leaching model with image data produced by taking 
an image of the piled same-shaped workpieces with 
the camera 

(4) For this selected teaching model, to detect the 



position and posture (direction) of the target work- 
piece for detection to the selected teaching model 
on the basis of the relative position and posture of 
each teaching model to the workpiece, stored in the 
5 above operation (2). 

[0006] The above device requires a plurality of teach- 
ing models generated by taking the image of the work- 
piece from the plurality of directions as shown in the 
10 above operation (1). 

[0007] It is an object of the present invention to pro- 
vide a device for generating a teaching model applied 
to an image processing device for detecting the three- 
dimensional position and posture of an object. 
is [0008] The present invention relates to a teaching 
model generating device for image processing, in which 
an object or an article having the same shape as that of 
the object is selected as a reference object, and three- 
dimensional position and/or posture of the object is rec- 

20 ognized by carrying out matching processing of a plu- 
rality of teaching models, which were generated and 
stored in advance on the basis of the respective image 
data produced by taking the image of the reference ob- 
ject from a plurality of directions, with the image data 

2S including the object. 

[0009] According to the first mode of the invention, 
one of the reference object and the image pickup means 
is fixed in place, while the other is fixed to a movable 
part of a robot or is grasped with a hand of the robot, 

30 and the robot is operated lor positioning to a plurality of 
image pickup positions where direction of the optical ax- 
is of the image pickup means with respect to the refer- 
ence object is different Irom one another respectively so 
that the image data respectively obtained at each of the 

35 image pickup positions is stored as a teaching model. 
[001 0] According to the second mode of the invention, 
the reference object is fixed to a movable part of a first 
robol or is grasped with a hand of the first robot, and an 
image pickup means is fixed to a movable part a second 

■*o robot or is grasped with a hand ol the second robot, and 
any one of or both of these first and second robots is 
operated for positioning to a plurality of image pickup 
positions where direction of the optical axis of the image 
pickup means with respect to the reference object is dif- 

45 ferent from one another respectively, so that the image 
data respectively obtained at each of said image pickup 
positions is stored as a teaching model. 
[0011] Preferably, the teaching model is a part of the 
image data of the reference object. 

so [0012] Preferably, the teaching model is composed of 
data obtained by performing a image processing on the 
image data of the reference object. 
[001 3] Preferably, the teaching model is generated for 
every direction in which the image pickup means took 

55 the image of the reference object and the teaching mod- 
el is stored in association with the information on the 
direction. 

[0014] Preferably, the image pickup means is a cam- 
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era. 

[0015] Preferably the image pickup means is a three 
dimensional visual sensor whose image pickup means 
measures the distance between the image pickup 
means and a plurality of points on the object. 
[0016] According to the present invention, the robot 
may be used to simply generate teaching models which 
are applied to an image processing device or the like for 
detecting the position and posture of an object through 
identification of the object regardless of three -dimen- 
sional variation in position and posture of the object. 
[0017] For a better understanding of the invention, 
and to show how the same may be carried into effect, 
reference will now be made, by way of example, to the 
accompanying drawings, in which:- 

Fig. 1 is view for explaining the outline of a first em- 
bodiment according to the present invention; 
Fig. 2 is a view for explaining the outline of a second 
embodiment according to the present invention; 
Fig. 3 is a view for explaining the outline of a third 
embodiment according to the present invention; 
Fig. 4 is a view showing teaching models in each of 
the above embodiments; 

Fig. 5 is a block diagram showing an essential part 
of a robot controller in each of the above embodi- 
ments; 

Fig. 6 is a block diagram showing an essential part 
of an image processing device in each of the above 
embodiments; 

Fig. 7 is a flow chart showing the procedure to gen- 
erate teaching models in the first embodiment; 
Fig. 8 is a flow chart showing the procedure to gen- 
erate teaching models in the second embodiment: 
Fig. 9 is a flow chart showing the procedure to gen- 
erate teaching models in the third embodiment; 
Fig. 1 0 is a flow chart showing the procedure of pick- 
ing job: 

Fig. 11 is a view for explaining the operation of a 
visual sensor for measurement of distance data for 
use in the embodiments according to the present 
invention; 

Fig. 1 2 is a view for explaining two-dimensional ar- 
ray data composed of distance data as image data 
produced by the above visual sensor for measure- 
ment of distance data; and 
Fig. 1 3 is a flow chart showing the procedure to pro- 
duce two-dimensional array data as the above im- 
age data. 

[0018] A teaching model generating device config- 
ured by a robot system will be described below as an 
embodiment according to the present invention. 
[0019] Fig. 1 is a schematic view showing the first em- 
bodiment for generation of a teaching model according 
to the present invention. As shown in Fig. 1 , a workpiece 
W is fixed in place to take an image of the workpiece W 
from a plurality of directions with an image pickup device 



20 grasped with a hand of a robot RB (or fixed to a mov- 
able part of the robot RB). A teaching model of an object 
(the workpiece W) is generated on the basis of image 
data produced by taking the image of the workpiece. For 
5 instance, four teaching models are respectively gener- 
ated on the basis of four image data produced by taking 
the image of the workpiece W from four directions as 
shown in Fig. 4. 

[0020] Fig. 5 is a block diagram showing an essential 

10 part of a controller of the robot RB in the present em- 
bodiment, and this controller has the same configuration 
as a robot controller in the prior art. 
[0021] As shown in Fig. 5, connected to a bus desig- 
nated by reference numeral 8 are a main processor 

'5 (which will be hereinafter simply referred to as a proc- 
essor) 1, a memory 2 composed of a RAM, a ROM and 
a non-volatile memory (EEPROM or the like), a teaching 
operation panel interface 3, an external device interface 
6, a communication interface 7 for communication with 

20 an image processing device and a servo control unit 5. 
A teaching operation panel 4 is connected to the teach- 
ing operation panel interface 3. 
[0022] A system program for supporting basic func- 
tions of Ihe robot and the robot controller is stored in the 

25 ROM of the memory 2. An operation program and its 
related set data of the robot taught according to an ap- 
plication are stored in the non-volatile memory of the 
memory 2. The RAM of the memory 2 is used for a stor- 
age area for temporary storage of data in various 

30 processing executed by the processor 1 . 

[0023] The servo control unit 5 is equipped with servo 
controllers 5a1 to 5an (n being a number resulting from 
adding the number of movable axes of a tool mounted 
to a robot wrist to Ihe tolal number of robot axes). Each 

35 servo controller is composed of a processor, a ROM and 
a RAM or the like to perlorm loop control ol position and 
speed of a servo motor (or driving each axis and also 
loop control of current. This servo controller constitutes 
a digital servo controller performing loop control of po- 

40 sition, speed and current in software. 

[0024] Output from the servo controllers 5a1 to 5an is 
provided through servo amplifiers 5b1 to 5bn to axis ser- 
vo motors M1 to Mn to control the drive ol these servo 
motors. Incidentally, although not shown, a position/ 

is speed detector is mounted to each of the servo motors 
M1 to Mn to detect the position and speed of each servo 
motor, and the detected position and speed are allowed 
to be fed back to the servo controllers 5a1 to 5an. 
[0025] The inputfoutput interface 6 is connected to 

50 sensors mounted to the robot, together with actuators 
and sensors of the peripheral equipment. 
[0026] Fig. 6 is a block diagram showing an image 
processing device 30 connected to an interface of the 
robot controller. The image processing device 30 has a 

55 processor 31, and the processor 31 is connected 
through a bus 40 to a ROM 32 for storage of a system 
program or ihe like executed by the processor 31, an 
image processor 33, an image pickup device interface 
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34 connected to the image pickup device, an MDI 35 
with display such as a CRT and a liquid crystal display 
for input and output of various commands and data, an 
image memory 36, a non-volatile memory 37, a RAM 38 
for temporary storage and so on of data and a commu- 
nication interface 39 connected to the robot controller 
10. 

[0027] An image produced by the image pickup de- 
vice 20 is stored in the image memory 26. The image 
processor 33 conducts processing of the image stored 
in the image memory according to a command from the 
processor 31 to recognize the object. The image 
processing device 30 is similar in configuration and 
function to an image processing device in the prior art. 
However, according to the present invention, the image 
processing device 30 is characterized in that teaching 
models (described later) are stored in the non-volatile 
memory 37, and pattern matching is carried out by using 
the stored teaching models, for the image data of the 
pile ot workpieces photographed by the image pickup 
device 20 to obtain three-dimensional position and pos- 
ture of the workpiece. 

[0028] As will be described later, the image pickup de- 
vice 20 is to produce the image data, and a CCD camera 
for producing a two-dimensional image or a visual sen- 
sor permitting the measurement of distance data is 
available for the image pickup device 20. The CCD cam- 
era produces the image data from the two-dimensional 
image according to a conventionally known method, 
while the visual sensor for measurement of distance da- 
ta is to produce, as image data, two-dimensional array 
data composed of distance data representing the dis- 
tance between the sensor and the object Since a three- 
dimensional visual sensor of spotlight scan type or the 
like disclosed in Japanese Patent Application Laid-open 
No. 7-270137 is well known as the visual sensor for 
measurement of distance data, a description will now 
be given of the three-dimensional visual sensor briefly. 
[0029] The three-dimensional visual sensor is adapt- 
ed for measuring the position of a beam spot irradiated 
onto the object with two scanners in arbitrary directions 
(X- and Y-directions) by using a detector (PSD) of posi- 
tion detection type. Thus, this three-dimensional visual 
sensor is effective in finding the three-dimensional po- 
sition of the object irradiated with the beam by calcula- 
tion on the basis of mirror deflection angles 8x, 6y of two 
scanners together with a light detecting position on the 
PSD. 

[0030] A description will now be simply given of a 
method of producing, as image data, two-dimensional 
array data composed of distance data with this three- 
dimensional visual sensor with reference to Figs. 11 to 
13. 

[0031] The range of scan over the object (the range 
of measurement) is determined in advance, and in this 
state, the three-dimensional visual sensor makes meas- 
urement of the three-dimensional position in each point 
on the X-Y plane in the predetermined range of scan to 



find the distance Z (i, j) in each point (i, j) between the 
sensor and the point of the beam irradiated onto the ob- 
ject in the manner of making a scan over the object in 
sequence of points (1, 1 ) to (1 , n), (2, 1) to (2, n), ... (m, 

5 1 ) to (m, n) on the above X-Y plane under discrete con- 
trol of the mirror deflection angles 8x, 6y of the scanners, 
as shown in Fig. 11. Then, this distance data Z (i, j) is 
stored in the RAM 38 of the image processing device 30. 
[0032] With the above operation, the three-dimen- 

10 sional visual sensor may produce, as image data, the 
two-dimensional array data composed of the distance 
data Z (i, j) between the sensor and the point of the beam 
irradiated onto the object, as shown in Fig. 12. 
[0033] Fig. 1 3 is a flow chart showing the processing 

' 5 performed by the processor 31 of the image processing 
device 30 to produce the image data. 
[0034] Firstly, indexes i, j are set at "1 " (Step 500) to 
issue a command to irradiate to the sensor 20, together 
with commanded mirror deflection angles e x, 8y corre- 

20 sponding to the start points y1, x1 (the point (1, 1) in Fig. 
11) in the predetermined range of measurement (Steps 
501 to 503). The sensor 20 irradiates opt ical beams after 
the mirrors are set at these deflection angles. Then, the 
sensor 20 outputs a signal produced by the PSD to the 

2 s image processing device 30. 

[0035] The processor 31 of the image processing de- 
vice 30 calculates the position of a beam spot formed 
on the object on the basis of the signal from the PSD 
and the commanded mirror deflection angles Ox, 8y to 

30 find the distance Z (i, j) between the sensor and the 
beam spot position, and then stores the distance data 
in the RAM 28 as two-dimensional array data [i, j) (Steps 
504 and 505) Instead of the processor 31, the sensor 
20 may be applied to calculation of finding the position 

35 of the beam spot formed on the object and also that of 
finding the distance Z (i, j) on the basis of the above 
beam spot position. 

[0036] Subsequently the processor 31 increments the 
index i by "1" to increase the mirror deflection angle 8x 

<*o in X-axial scanning by a predetermined amount Ax 
(Steps 506, 507), and decides whether or not the index 
i exceeds a set value n (Step 508). If not exceeding, the 
processing is returned to Step 503 to find the distance 
Z (i, j) in the next point by repeating the processing in 

is Steps 503 to 508. Subsequently, the processing in 
Steps 503 to 508 is executed for finding and storing the 
distance Z (i, j) at each point from (1, 1) to (1, n) in Fig. 
11 until the index i exceeds the set value n. 
[0037] When it is decided in Step 508 that the index i 

50 exceeds the set value n, the processor 31 sets the index 
i at "1 ", and increments the index j by "1" to increase the 
mirror deflection angle 8y in Y-axial scanning by a pre- 
determined amount Ay (Steps 509 to 51 1 ). Subsequent- 
ly, the processor 31 decides whether or not the index j 

55 exceeds a set value m (Step 512). If not exceeding, the 
processing is returned to Step 502 to repeat the 
processing in Steps 502 to 512. 
[0038] As described above, the processing in Steps 
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502 to 512 is performed repeatedly until the index j ex- 
ceeds the set value m. The measurement ot all the 
points in the range of measurement (the range of scan) 
shown in Fig. 11 is considered to be finished as of the 
time when the index j exceeds the set value m. Then, 
the distance dataZ(1, 1)toZ(m, n) serving as the two- 
dimensional array data is stored in the RAM 28, resulting 
in the completion of the processing to produce the image 
data. 

[0039] The above processing is applied to produce 
the two-dimensional array data as the image data with 
the visual sensor for measurement of distance data. 
[0040] The two-dimensional array data produced as 
described above may be used as the image data to gen- 
erate teaching models. However, for simplification, a de- 
scription will now be given of the procedure to generate 
teaching models on the basis of the image data pro- 
duced by taking in the image of the object with the CCD 
camera 20 serving as an image pickup device for taking 
in image data. 

[0041] A description will be given of the procedure to 
generate teaching models by using the teaching model 
generating device according to the present embodiment 
configured by the robot controller 10 and the image 
processing device 30 with reference to the flow chart 
shown in Fig. 7. 

[0042] An operator sets a work coordinate system as 
to the workpiece fixed in place in the robot controller 10 
through Ihe teaching operation panel 4. The work coor- 
dinate system is stored in the robot controller (Step 100). 
In this place, the meaning of setting of the work coordi- 
nate system is that a coordinate transformation matrix 
of the work coordinate system as seen from a world co- 
ordinate system provided in the robot is set in Ihe robot 
controller. 

[0043] Then, calibration of the camera mounted to the 
robot wrist is performed and camera mounting position 
and posture based on the center of the flange surface 
of the robot wrist is set in the robot control device 10. 
As a result, a camera coordinate system is set in the 
robot (Step 101). 

[0044] Then, for specifying the first (0 th ) position and 
posture of the camera mounted to the end of the robot 
wrist for the image-taking operation with respect to the 
workpiece W and subsequent positions and postures, a 
turning axis and a turning angle for turning the camera 
from the first position and posture is set, and also the 
number N of positions and postures which the camera 
assumes for the image-taking operation is set (Step 
103). 

[0045] For instance, four teaching models are gener- 
ated on the basis of four image data produced by taking 
the image of the workpiece W from four directions (N=4) 
as shown in Fig. 4. 

[0046] Fig. 4(a) shows a first teaching model gener- 
ated through the image-taking operation with the cam- 
era in the 0 ,h position and posture. The first teaching 
model is generated on the basis of the image data pro- 



duced by taking the image of the top of the workpiece 
W with the camera from the Z-axis direction of the world 
coordinate system. 

[0047] Subsequently, the robot is operated to turn the 
s camera by a certain angle about an axis, which passes 
through a specific point (an origin of the work coordinate 
system) on the workpiece W and is perpendicular to the 
center axis of the camera, to the first position and pos- 
ture for the image-taking operation. Incidentally when 

10 ihe Z-axis of the world coordinate system is set to be 
parallel to the center axis of the camera in the first (0 th ) 
position and posture, the first position and posture next 
to the 0 lh position and posture for the image-taking op- 
eration assume to be the position and posture provided 

is by turning the camera by a certain angle (for instance, 
30 degrees) about an axis parallel to the axis (for in- 
stance, the X-axis) perpendicular to the Z-axial direction 
as shown in Fig. 4(b). Then, the second teaching model 
is generated on the basis of the image data (see Fig. 4 

20 (b)) produced by taking the image of the workpiece W 
with the camera in the above first position and posture. 
[0048] Thereafter, in the same way the camera is 
caused to turn by 60 and 90 degrees respectively from 
the position shown in Fig. 4(a) about the axis, which 

25 passes through the specific point on the workpiece and 
is parallel to the X-axis, to the second and third positions 
and postures for the image-taking operation to take the 
image of the workpiece W respectively. Then, the third 
and fourth teaching models are generated on the basis 

30 of the imagedata (see Figs. 4(c) and 4(d)) obtained from 
the images the camera look at the respective positions 
and postures. 

[0049] In this place, a description will be given of one 
example of generating four teaching models in the 

35 above manner with respect to the workpiece serving as 
the object and application of these models. Incidentally 
in the above embodiment, camera is caused to turn by 
0, 30, 60 and 90 degrees about the workpiece serving 
as the object to generate four teaching models by taking 

40 the image of the workpiece with the camera from the 
respective positions. Alternatively, if the turning angle of 
the robot to turn the camera is set at a smaller pitch for 
generation of more teaching models, it is possible to de- 
tect Ihe three-dimensional position and posture of the 

45 workpiece with higher accuracy. 

[0050] As described above, the position and posture 
of the robot where the first (0 lh ) image-taking operation 
is performed with the camera 20 mounted to the end of 
the robot wrist are taught together with the axis serving 

50 as the center axis of turning and the turning angle, and 
the number N of positions and postures for the image- 
taking operation is set. For easy understanding, it is as- 
sumed that the position in which the center axis of the 
camera is parallel to the Z-axis of the world coordinate 

55 system and which has the same coordinate values of 
the X- and Y-axes, other than the Z-axis, on the world 
coordinate system as those of the workpiece W put in 
predetermined position and posture is taught as the po- 
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sition and posture for generating the 0 lh teaching model. 
Further, the positions turned by 30, 60 and 90 degrees 
about the axis, which passes through the position ot the 
workpiece W and is parallel to the X-axis oJ the world 
coordinate system, are set as the first, second and third 
positions and postures for the image-taking operation. 
It is also assumed that the number N of positions for the 
image-taking operation is set at "4". 
[0051] When a command to generate the teaching 
model is supplied through the teaching operation panel 
4, the processor 1 of the robot controller 10 initializes a 
counter M for counting the image-taking times to "0" 
(Step 103), and outputs a command to take the image 
with the camera to the image processing device 30 after 
operating the robot to move the camera to the Mth (=0) 
position and posture (Step 104). The image processing 
device 30, in response to the above command, causes 
the camera to take the image of the workpiece W and 
stores the image data in the image memory 36. In addi- 
tion to the above operation, the image processing de- 
vice 30 generates the Mth teaching model on the basis 
of the image data and stores the generated Mth teaching 
model in the non-volatile memory 37 (Step 105), 
[0052] Further, 1he image processing device 30 cal- 
culates the position and posture of the work coordinate 
system as a coordinate transformation matrix as seen 
from the camera coordinate system to find the relative 
position and posture of the camera to the workpiece, 
and then stores the found relative position and posture 
in the non-volatile memory 37 as the relative position 
and posture of the Mih teaching model and outputs a 
data acquisition signal to the robot controller (Step 106). 
For instance, the above relative position and posture are 
stored in the form of [xO, yO, zO, ccO, [30, yO,] c as the 
position and posture of the camera coordinate system, 
where a, p and y represent the turning angles about the 
X-, Y- and Z-axes, respectively and "C" represents the 
camera coordinate system 

[0053] Subsequently, the processor 1 of the robot 
controller 10 increments the counter M by "1" in re- 
sponse to the data acquisition signal (Step 107), and 
decides whether or not the value of the counter M is 
smaller than the set value N (=4) (Step 108). When the 
counter value is smaller than the set value, Ihe process- 
ing is returned to Step 104 to move the robot to the Mth 
position and posture. That is, in the above embodiment 
shown in Fig. 4, the cam is caused to turn by 30 degrees 
about the axis, which passes through the position of the 
workpiece and is parallel to the X-axis of the world co- 
ordinate system, to store the leaching model and the ■ 
relative position and posture of the camera to the work- 
piece at that time. 

[0054] In the following operation, the processing in 
Steps 104 to 108 is executed for storing the generated 
teaching model and the relative position and posture of t 
the camera to the workpiece in the non-volatile memory 
until the value of the counter M reaches the set value N 
(=4). In the above example, the teaching models to be 



stored are those produced from the image of Figs. 4(a) 
to 4(d). For the above four teaching models, the relative 
position and posture of the camera to the workpiece are 
stored in the form of [xO, yO, zO, ctO, po, y0]c, [x1, y1, z1, 
5 <x1, P1, ?1]c, [x2, y2, z2, cc2, P2, 72]cand [x3, y3, z3, 3a, 
P3. y3]c as the positions and postures of the workpiece 
W of the camera coordinate system. 
[0055] With the above operation, Ihe leaching model 
and the relative position of the camera to the workpiece 
10 are stored in the non-volatile memory 37 of the image 
processing device 30. 

[0056] Fig. 2 is a schematic view showing the second 
embodiment according to the present invention. The 
second embodiment is different from the first embodi- 
« ment shown in Fig. 1 in that the camera 20 is fixed in 
place, and Ihe workpiece W serving as the object is 
grasped with the hand mounted to the end of the robot 
wrist or is fixed to the movable part of the robot for gen- 
eration and storage of the teaching modeL The robot 
20 controller 10 and the image processing device 30 in the 
second embodiment are similar in configuration to those 
in the first embodiment, except for the processing for 
generation of the teaching modeL Fig. 8 shows the flow 
of the procedure to generate the teaching modeL 
55 [0057] After calibration of the camera 20 fixed in 
place, the camera coordinate system is set in the robot 
controller 10 and stored in the memory 2 (Step 200). 
Accordingly, the coordinale transformation matrix of ihe 
camera coordinate system as seen from the world co- 
30 ordinate system may be produced. 

[0058] Then, Ihe operator sets, in the robot controller 
10, the position and posture of the robot wrist, as seen 
from the center ot the flange surface of the robot wrist 
in the coordinate system (the work coordinate system) 
as supposed on the workpiece grasped with the robot hand 
(or fixed to the movable part of the robot) are measured 
and the measured data is set in the robot controller 10 
and stored in the memory 2. As a result, the robot is 
allowed to recognize the position and posture in the co- 
io ordinate system (the work coordinate system) supposed 
on the workpiece when the robot is at any position. 
[0059] Then, similarly to the first embodiment, the op- 
erator sets the first (0 ,h ) position and posture of the work- 
piece W grasped with the robot hand for the image-tak- 
15 ing operation with the camera 20 fixed in place, together 
with the turning axis and turning angle of the robot to 
turn the workpiece with respect to the camera 20 fixed 
in place from the first position and posture in order to 
specify the positions and postures for the subsequent 
50 image-taking operations (Step 202). 

[0060] The positions and postures resulting from turn- 
ing Ihe workpiece grasped with the robot hand by 30, 
60 and 90 degrees about the axis passing through the 
camera 20 and being parallel to the X-axis of the work 
•s coordinate system are set as the first, second and third 
positions for the image-taking operation in the similar 
manner to the first embodiment. As long as Ihe number 
N of positions for the image-taking operation is set at 
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"4", it is possible to generate the similar teaching models 
to the first embodiment as shown in Fig. 4. 
[0061] When a command to generate the teaching 
model is supplied through the teaching operation panel 
4, the processor 1 of the robot controller 10 initializes 
the counter M to "0", operates the robot so that the work- 
piece W moves to the Mth (=0) position and posture and 
issues a command to take the image with the camera 
20 to the image processing device 30 (Steps 203, 204) 
[0062] The image processing device 30 takes the im- 
age of the workpiece W with the camera in response to 
the above command and generates the Mth teaching 
model on the basis of the image data produced from the 
image, and stores the generated teaching models in the 
non-volatile memory 37. Further, the image processing 
device 30 calculates the position and posture of the co- 
ordinate system (the work coordinate system) supposed 
on the workpiece as the coordinate transformation ma- 
trix as seen from the camera coordinate system, stores 
the calculated position and posture in the non-volatile 
memory 37 as the Mth teaching model for the relative 
position and posture of the camera to the workpiece and 
outputs a data acquisition signal to the robot controller 
(Steps 204 to 206). 

[0063] The processor 1 of the robot controller 1 incre- 
ments the counter M by "1 " in response to the data ac- 
quisition signal (Step 207), and decides whether or not 
the value of the counter M is smaller than the set value 
N (=4). When the value of the counter M is smaller than 
the set value N, the processing is returned to Step 204 
to move the robot to the Mth position and posture. 
[0064] In the following operation, the processing in 
Steps 204 to 208 is executed for storing the teaching 
model and the relative position and posture of the cam- 
era to the workpiece in the non-volatile memory 37 in 
the same way as in the first embodiment until the value 
of the counter M reaches the set value N (=4). 
[0065] Fig. 3 is a schematic view showing the third 
embodiment according to the present invention. Accord- 
ing to the third embodiment, both the workpiece and the 
camera are respectively fixed to the movable parts of 
the robots or are grasped with the hands thereof for gen- 
eration of the teaching model. The robot controllers of 
robots RB1 , RB2 in the third embodiment are substan- 
tially similar in configuration to that shown in Fig. 5, ex- 
cept that two robot controllers are connected together 
through the communication interface and that the image 
processing device 30 is connected to only the second 
robot controller. The image processing device 30 is also 
similar in configuration to that shown in Fig. 6. 
[0066] The workpiece W serving as the object is 
grasped with the hand of the first robot RB1 and the 
camera 20 is mounted to the wrist of the second robot 
RB2. The position and posture of the second robot RB2 
in the world coordinate system as seen from the world 
coordinate system of the first robot RB1 is measured to 
find the coordinate transformation matrix (A), which is 
then stored in the first robot controller (Step 300). 



[0067] Further, the coordinate transformation matrix 
(B) of the position and posture of the coordinate system 
(the work coordinate system) supposed on the work- 
piece W grasped with the hand of the first robot RB1 , as 
5 seen from the center of the flange surface of the robot 
wrist, are set and stored in the robot controller of the first 
robot RB1 (Step 301). Calibration of the camera 20 
mounted to the wrist of the second robot RB2 is per- 
formed and the transformation matrix (C) from the cent- 
10 er of the flange surface of the second robot wrist to the 
camera coordinate system is set and stored in the robot 
controller of the first robot RB1 (Step 302). 
[0068] Further, like the first and second embodiments, 
the position and posture of each of the first and second 

is robots are set in each of the robot controllers such that 
the camera 20 and the workpiece W of which image the 
camera 20 first takes to assume the first (0 ,h ) relative 
position and posture for the image-taking operation. 
Subsequently the first, second,... and ith positions and 

20 postures for the image-taking operation are set together 
with the number N of positions and postures for the im- 
age-taking operation. According to the third embodi- 
ment, the relative position and posture of the camera to 
the workpiece are set by turning any one or both of the 

25 camera and the workpiece by a predetermined turning 
angle relatively to each other in a manner such that the 
second robot RB2 mounted with the camera is operated 
to turn the camera relatively to the workpiece W without 
shifting the position and posture of the first robot RB1 

30 grasping the workpiece W from the first position and 
posture, or that the first robot RB1 is operated to turn 
the workpiece W while the camera 20 is fixed in place 
without shifting the position and posture of the second 
robot RB2. 

35 [0069] Further, the turning axis of each of the robots 
is set to be an axis which is orthogonal to the center axis 
of the camera 20 parallel to an axis of each world coor- 
dinate system and also passes through the center po- 
sition (the origin of the camera coordinate system or the 

to origin of the work coordinate system) of the other party. 
For instance, in the embodiment shown in Fig. 3, the 
angles the workpiece W or camera 20 turns about Z axis 
(vertical axis) ot each of the world coordinate system 
which passes through the camera 20 or the workpiece 

is W is set. For simple setting of the turning axis and turn- 
ing angle, one of the workpiece W and the camera 20 
is turned with the other fixed in place. However, in the 
case where the robot operated for turning the workpiece 
or the camera is prevented from moving, the other robot 

50 for keeping the camera or the workpiece fixed is driven 
such that the camera and the workpiece assume the rel- 
ative positron and posture set in advance (Step 303). 
[0070] When the command to generate the leaching 
model is supplied through the teaching operation panel 

>s 4 any of one of the robot controllers, the processor 1 of 
the robot controller 10 of each of the robots RB1.RB2 
initializes the counter M to "0° operates the robot to 
move the workpiece W to the Mth (0 th ) position and pos- 
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ture and issues a command to take the image with the 
camera 20 to the image processing device 30 (Steps 
304, 305). The image processing device 30 takes the 
image of the workpiece W with the camera in response 
to the above command, generates the Mth teaching 
model on the basis of the image data produced from the 
image and stores the generated teaching model in the 
non-volatile memory 37 (Step 306). 
[0071] Further, the position and posture of the coordi- 
nate system supposed on the workpiece W is obtained 
as the coordinate transformation matrix (D) as seen 
from the world coordinate system of the first robot, on 
the basis of the position and posture of the first robot as 
of the time of image-taking operation and also the coor- 
dinate transformation matrix (B) set in Step 301 In ad- 
dition, the position and posture as seen from the camera 
coordinate system is obtained as the coordinate trans- 
formation matrix (E) as seen from the world coordinate 
system of the second robot, on the basis of the position 
and posture of the second robot and also the coordinate 
transformation matrix (C) set in Step 302 (Step 307) 
The position and posture supposed on the workpiece W 
is transformed into the coordinate transformation matrix 
(F) as seen from the camera coordinate system through 
the operation of the lollowing expression (1 ) on the basis 
of the coordinate transformation matrices (D), (E) found 
as described above and also the coordinate transforma- 
tion matrix (A) set in Step 300, and is stored as the Mlh 
relative position and posture, and then the data acqui- 
sition signal is outputted to both the robot controllers. 

(F) = (E)"' (A)" 1 (D) (1) 

[0072] The processor 1 of each robot controller 1 0 in- 
crements the counter M by "1 " in response to the data 
acquisition signal (Step 308), and decides whether or 
not the value of the counter M is smaller than the set 
value N (=4) (Step 309). When the value of the counter 
M is smaller than the set value, the processing is re- 
turned to Step 305 to move the robot to the Mth position 
and posture. 

[0073] In the following operation, the processing in 
Steps 305 to 310 is executed for storing the teaching 
model and the relative position and posture of the cam- 
era 20 to the workpiece W in the non-volatile memory 
37 in the same manner as in the above first and second 
embodiments until the value of the counter M reaches 
the set value N. 

[0074] In addition to the above three kinds of embod- 
iments for generation of the teaching models, a descrip- 
tion will now be given of the application of the teaching 
models to the case of starting a picking job to pick out 
the individual workpiece from the pile of workpieces W, 
for instance. 

[0075] The robot wrist is mounted with the camera to- 
gether with the hand. When a command to pick a work- 



piece is supplied through the teaching operation panel 
or the like to the robot controller 10, the processor 1 
starts the procedure of the picking job as shown in Fig. 
10. 

5 [0076] Alter operating the robot to move the camera 
mounted to the wrist of the taught robot to the image 
pickup position where the piled workpieces are in the 
field of vision of the camera, the processor 1 firstly out- 
puts the three-dimensional position and posture of the 
'0 camera on the world coordinate system to the image 
processing device 30 together with a command to take 
the image of the workpiece (Steps 400, 401). 
[0077] The processor 31 of the image processing de- 
vice 30 lakes the image of the pile of workpiece in re- 
's sponse to the image pickup command, data obtains the 
image data of several workpieces, and stores in the im- 
age memory 36 (Step 402). 

[0078] Subsequently, the processor 31 detects a 
workpiece by performing pattern matching by using one 

20 (ihe 0 th teaching model) of the teaching models stored 
in the non-volatile memory 37 (Step 403). In the above 
pattern matching, variations in position, turning and 
scale ol the image of the workpieces are detected. Then, 
the processor 31 decides whether or not the delected 

25 workpiece has a matching value of not less than a ref- 
erence value (Step 404). When the workpiece having 
the matching value of not less than the reference value 
is nol lound, the processor 31 further decides whether 
or not the pattern matching was completed for all the 

30 teaching models (the 0 lh to 3* teaching models) (Step 
405). If not completed yet, the processor 31 makes the 
pattern matching of the image data with the other teach- 
ing model (Step 406). 

[0079] When the image data of the workpiece having 

35 the matching value ol not less than the certain reference 
value is detected through the pattern matching of the 
image data with any of the teaching models in Step 404, 
pattern matching of the detected image data of the work- 
piece is carried out with all the other teaching models. 

40 That is, the processor 31 carries out the pattern match- 
ing of the detected image data of the workpiece with all 
the stored teaching models (Step 407). 
[0080] The processor 31 selects the teaching model 
of the highest matching value obtained by the above pat- 

15 tern matching to obtain the ultimate relative position and 
posture of the camera to the workpiece on the basis of 
the relative position and posture of the camera to the 
workpiece stored in association with the selected teach- 
ing model and also the amount of variations in position, 

50 turning and scale within the image in the pattern match- 
ing of the image data with respect to the selected teach- 
ing model. 

[0081] Then, the processor 31 finds Ihe position and 
posture of the workpiece on the world coordinate system 
55 on the basis of the above ultimate relative position and 
posture and also the position and posture of the camera 
in the world coordinate system sent through Ihe 
processing in Step 401 and outputs the above found po- 
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sition and posture. That is, since the relative position 
and posture of the workpiece to the camera is the posi- 
tion and posture of the workpiece as seen from the cam- 
era coordinate system, the position and posture of the 
detected workpiece on the work coordinate system may s 
be found through operation of coordinate transformation 
on the basis of the data of the above position and pos- 
ture and also the data of the position and posture of the 
camera in the world coordinate system (Step 409) 
[0082] The robot controller 10 actuates the robot on 10 
the basis of the transmitted three-dimensional position 
and posture of the detected workpiece W, picks the de- 
tected workpiece and move the workpiece to a prede- 
termined taught position in the manner similar to the pri- 
or art (Step 410). Then, the processing is returned to '5 
Step 402 to repeat the processing from Step 402. 
[0083] When all the workpieces are picked from the 
pile of workpieces, a matching value of not less than the 
certain reference value can not be detected any more 
by conducting the pattern matching of the image data 20 
with all the teaching models in the processing in Steps 
403 to 406, so that the picking job is finished. 



Claims ss 

1. A teaching model generating device for image 
processing, in which an object or an article having 
the same shape as that of the object is selected as 
a reference object, and three-dimensional position 30 
and/or posture of said object is recognized by car- 
rying out matching processing of a plurality of teach- 
ing models, which were generated and stored in ad- 
vance on the basis of the respective image data pro- 
duced by taking the image of said reference object 3S 
from a plurality of directions, with the image data 
including the object, 

wherein one of the reference object and the 
image pickup means is fixed in place, while the oth- 
er is fixed to a movable part of a robot or is grasped to 
with a hand of the robot, and said robot is operated 
for positioning to a plurality of image pickup posi- 
tions where direction of the optical axis of said im- 
age pickup means with respect to the reference ob- 
ject is different from one another respectively, so is 
that the image data respectively obtained at each 
of said image pickup positions is stored as a teach- 
ing modeL 

2. A teaching model generating device for image so 
processing, in which an object or an article having 
the same shape as that of the object is selected as 
a reference object, and three- dimensional position 
and/or posture of said object is recognized by car- 
rying out matching processing of a plurality of teach- 55 
ing models, which were generated and stored in ad- 
vance on the basis of the respective image data pro- 
duced by taking the image of said reference object 



from a plurality of directions, with the image data 
including the object, 

wherein the reference object is fixed to a mov- 
able part of a first robot or is grasped with a hand 
of the first robot, and an image pickup means is 
fixed to a movable part a second robot or is grasped 
with a hand of the second robot, and any one of or 
both of said first and second robots is operated for 
positioning to a plurality of image pickup positions 
where direction of the optical axis of said image 
pickup means with respect to the reference object 
is different from one another respectively, so that 
the image data respectively obtained at each of said 
image pickup positions is stored as a teaching mod- 
el. 

3. A leaching model generating device according to 
claims 1 or 2, wherein said teaching model is a part 
of the image data of the reference object. 

4. A teaching model generating device according to 
any preceding claim, wherein said teaching model 
is composed of data obtained by performing a im- 
age processing on the image data of the reference 

5. A teaching model generating device according to 
any preceding claim, wherein said teaching model 
is generated for every direction in which said image 
pickup means took the image of said reference ob- 
ject and said teaching model is stored in association 
with the information on the direction. 

6. A leaching model generating device according to 
any preceding claim, wherein said image pickup 
means is a camera. 

7. A leaching model generating device according to 
any preceding claim, wherein said image pickup 
means is a three-dimensional visual sensor whose 
image pickup means measures the distance be- 
tween the image pickup means and a plurality of 
points on the object. 
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